Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1960 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.4806. Min Act: -0.0001

Data Index: 1498443 (Python Code)

Max Activating Token Index: 617

Text #1

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.5777. Min Act: -0.0001

Data Index: 1641869 (Python Code)

Max Activating Token Index: 935

Text #2

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.3012. Min Act: -0.0001

Data Index: 1445912 (Python Code)

Max Activating Token Index: 974

Text #3

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0720. Min Act: -0.0001

Data Index: 1482866 (Python Code)

Max Activating Token Index: 1013

Text #4

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.3955. Min Act: -0.0001

Data Index: 1383921 (Python Code)

Max Activating Token Index: 867

Text #5

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0588. Min Act: -0.0001

Data Index: 1592803 (Python Code)

Max Activating Token Index: 625

Text #6

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0247. Min Act: -0.0001

Data Index: 1463268 (Python Code)

Max Activating Token Index: 751

Text #7

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0047. Min Act: -0.0001

Data Index: 1534946 (Python Code)

Max Activating Token Index: 663

Text #8

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.9453. Min Act: -0.0001

Data Index: 1550174 (Python Code)

Max Activating Token Index: 749

Text #9

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.4519. Min Act: -0.0001

Data Index: 1509084 (Python Code)

Max Activating Token Index: 713

Text #10

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0654. Min Act: -0.0001

Data Index: 1544582 (Python Code)

Max Activating Token Index: 443

Text #11

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.9434. Min Act: -0.0001

Data Index: 697588 (C4 (Web Text))

Max Activating Token Index: 787

Text #12

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.3002. Min Act: -0.0001

Data Index: 1527447 (Python Code)

Max Activating Token Index: 673

Text #13

Max Range: 5.5777. Min Range: -5.5777

Max Act: 5.0063. Min Act: -0.0001

Data Index: 1620427 (Python Code)

Max Activating Token Index: 895

Text #14

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.9877. Min Act: -0.0001

Data Index: 1587309 (Python Code)

Max Activating Token Index: 799

Text #15

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.8220. Min Act: -0.0001

Data Index: 1530506 (Python Code)

Max Activating Token Index: 246

Text #16

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.8266. Min Act: -0.0001

Data Index: 1516210 (Python Code)

Max Activating Token Index: 587

Text #17

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.6341. Min Act: -0.0001

Data Index: 1537324 (Python Code)

Max Activating Token Index: 278

Text #18

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.8094. Min Act: -0.0001

Data Index: 1461207 (Python Code)

Max Activating Token Index: 575

Text #19

Max Range: 5.5777. Min Range: -5.5777

Max Act: 4.7950. Min Act: -0.0001

Data Index: 1397232 (Python Code)

Max Activating Token Index: 353