Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3852 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.6320. Min Act: -0.0000

Data Index: 1406450 (Python Code)

Max Activating Token Index: 876

Text #1

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.5570. Min Act: -0.0000

Data Index: 1481617 (Python Code)

Max Activating Token Index: 84

Text #2

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.3841. Min Act: -0.0000

Data Index: 1408697 (Python Code)

Max Activating Token Index: 986

Text #3

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.2562. Min Act: -0.0000

Data Index: 1470991 (Python Code)

Max Activating Token Index: 937

Text #4

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.3396. Min Act: -0.0000

Data Index: 1385842 (Python Code)

Max Activating Token Index: 649

Text #5

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.4451. Min Act: -0.0000

Data Index: 1607036 (Python Code)

Max Activating Token Index: 466

Text #6

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.2675. Min Act: -0.0000

Data Index: 1386990 (Python Code)

Max Activating Token Index: 653

Text #7

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.2275. Min Act: -0.0000

Data Index: 1577670 (Python Code)

Max Activating Token Index: 494

Text #8

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.2035. Min Act: -0.0000

Data Index: 1451068 (Python Code)

Max Activating Token Index: 530

Text #9

Max Range: 2.6320. Min Range: -2.6320

Max Act: 1.9899. Min Act: -0.0000

Data Index: 1526230 (Python Code)

Max Activating Token Index: 368

Text #10

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0005. Min Act: -0.0000

Data Index: 1386118 (Python Code)

Max Activating Token Index: 261

Text #11

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0983. Min Act: -0.0000

Data Index: 1555740 (Python Code)

Max Activating Token Index: 558

Text #12

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.3106. Min Act: -0.0000

Data Index: 1594005 (Python Code)

Max Activating Token Index: 847

Text #13

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0661. Min Act: -0.0000

Data Index: 1407867 (Python Code)

Max Activating Token Index: 715

Text #14

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.2957. Min Act: -0.0000

Data Index: 1606410 (Python Code)

Max Activating Token Index: 379

Text #15

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0335. Min Act: -0.0000

Data Index: 1523160 (Python Code)

Max Activating Token Index: 343

Text #16

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0392. Min Act: -0.0000

Data Index: 1618062 (Python Code)

Max Activating Token Index: 579

Text #17

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.1444. Min Act: -0.0000

Data Index: 1615919 (Python Code)

Max Activating Token Index: 682

Text #18

Max Range: 2.6320. Min Range: -2.6320

Max Act: 2.0322. Min Act: -0.0000

Data Index: 1499537 (Python Code)

Max Activating Token Index: 809

Text #19

Max Range: 2.6320. Min Range: -2.6320

Max Act: 1.8810. Min Act: -0.0000

Data Index: 1367295 (Python Code)

Max Activating Token Index: 860