The AMD64 manual does not mention this case something the hardware will do an implicit invalidation of. It seems that just having PDE.PS bit different should trigger a TLB miss and the hardware will walk the page table and replace the TLB entry with the new one. I can understand the intention though I am not sure if the implementation should be done in this way. A more accurate implementation seems to be to update the TLB::lookup function such that it does not return the entry when the logBits are different.

This solution seems very sketchy to me.  I would not be surprised if the current code is broken, but I doubt this is the proper fix.  Although I don't see where the AMD manual says it explicitly, I expect that the result of adding a new mapping at a larger size without invalidating possible smaller mappings out of the TLB first would be undefined behavior.  I'll claim that, if you are really seeing two conflicting mappings being added to the TLB, then either (1) there's a prior TLB invalidation that wasn't handled properly or (2) the OS is broken.  I'd put my money on the former.  I noticed that there are several cases that are supposed to implicitly invalidate the entire TLB (from section 5.5.2 "TLB Management" of Vol. 2 of the AMD64 manual set).  I wonder if we are missing some of these?
TLB entries can be explicitly invalidated using operations intended for that purpose or implicitly

invalidated as a result of another operation. TLB invalidation has no effect on the associated page-translation

tables in memory.
Explicit Invalidations. Three mechanisms are provided to explicitly invalidate the TLB:

• The invalidate TLB entry instruction (INVLPG) can be used to invalidate specific entries within

the TLB. This instruction invalidates a page, regardless of whether it is marked as global or not.

The Invalidate TLB entry in a Specified ASID (INVLPGA) operates similarly, but operates on the

specified ASID. See “Invalidate Page, Alternate ASID” on page 474.

• Updates to the CR3 register cause the entire TLB to be invalidated except for global pages. The

CR3 register can be updated with the MOV CR3 instruction. CR3 is also updated during a task

switch, with the updated CR3 value read from the TSS of the new task.

• The TLB_CONTROL field of a VMCB can request specific flushes of the TLB to occur when the

VMRUN instruction is executed on that VMCB. See “TLB Flush” on page 473.
Implicit Invalidations. The following operations cause the entire TLB to be invalidated, including

global pages:

• Modifying the CR0.PG bit (paging enable).

• Modifying the CR4.PAE bit (physical-address extensions), the CR4.PSE bit (page-size

extensions), or the CR4.PGE bit (page-global enable).

• Entering SMM as a result of an SMI interrupt.

• Executing the RSM instruction to return from SMM.

• Updating a memory-type range register (MTRR) with the WRMSR instruction.

• External initialization of the processor.

• External masking of the A20 address bit (asserting the A20M# input signal).

• Writes to certain model-specific registers with the WRMSR instruction; see the BIOS and Kernel

Developer's Guide applicable to your product for more information