Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2048 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.1946. Min Range: -4.1946

Max Act: 4.1069. Min Act: -0.0000

Data Index: 1368420 (Python Code)

Max Activating Token Index: 252

Text #1

Max Range: 4.1946. Min Range: -4.1946

Max Act: 4.1191. Min Act: -0.0000

Data Index: 1496696 (Python Code)

Max Activating Token Index: 433

Text #2

Max Range: 4.1946. Min Range: -4.1946

Max Act: 4.0905. Min Act: -0.0000

Data Index: 1376626 (Python Code)

Max Activating Token Index: 316

Text #3

Max Range: 4.1946. Min Range: -4.1946

Max Act: 4.0601. Min Act: -0.0000

Data Index: 1465679 (Python Code)

Max Activating Token Index: 558

Text #4

Max Range: 4.1946. Min Range: -4.1946

Max Act: 4.1946. Min Act: -0.0000

Data Index: 1405556 (Python Code)

Max Activating Token Index: 681

Text #5

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.6394. Min Act: -0.0000

Data Index: 1573153 (Python Code)

Max Activating Token Index: 647

Text #6

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.8873. Min Act: -0.0000

Data Index: 1446990 (Python Code)

Max Activating Token Index: 680

Text #7

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.7333. Min Act: -0.0000

Data Index: 1457675 (Python Code)

Max Activating Token Index: 818

Text #8

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.5206. Min Act: -0.0000

Data Index: 1597173 (Python Code)

Max Activating Token Index: 991

Text #9

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.8076. Min Act: -0.0000

Data Index: 1620070 (Python Code)

Max Activating Token Index: 697

Text #10

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.5129. Min Act: -0.0000

Data Index: 1544411 (Python Code)

Max Activating Token Index: 506

Text #11

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.3369. Min Act: -0.0000

Data Index: 1394662 (Python Code)

Max Activating Token Index: 713

Text #12

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.1261. Min Act: -0.0000

Data Index: 1558214 (Python Code)

Max Activating Token Index: 397

Text #13

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.3605. Min Act: -0.0000

Data Index: 1536561 (Python Code)

Max Activating Token Index: 783

Text #14

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.0730. Min Act: -0.0000

Data Index: 1620264 (Python Code)

Max Activating Token Index: 740

Text #15

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.3805. Min Act: -0.0000

Data Index: 1572587 (Python Code)

Max Activating Token Index: 205

Text #16

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.0170. Min Act: -0.0000

Data Index: 1630347 (Python Code)

Max Activating Token Index: 368

Text #17

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.4112. Min Act: -0.0000

Data Index: 1362313 (Python Code)

Max Activating Token Index: 217

Text #18

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.1016. Min Act: -0.0000

Data Index: 1387864 (Python Code)

Max Activating Token Index: 767

Text #19

Max Range: 4.1946. Min Range: -4.1946

Max Act: 3.2344. Min Act: -0.0000

Data Index: 1505828 (Python Code)

Max Activating Token Index: 364