Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3958 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.2138. Min Range: -5.2138

Max Act: 5.2138. Min Act: -0.0000

Data Index: 1581989 (Python Code)

Max Activating Token Index: 905

Text #1

Max Range: 5.2138. Min Range: -5.2138

Max Act: 5.1313. Min Act: -0.0000

Data Index: 1380690 (Python Code)

Max Activating Token Index: 677

Text #2

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.7915. Min Act: -0.0000

Data Index: 1390384 (Python Code)

Max Activating Token Index: 718

Text #3

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.4474. Min Act: -0.0000

Data Index: 1398308 (Python Code)

Max Activating Token Index: 281

Text #4

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.2869. Min Act: -0.0000

Data Index: 276968 (C4 (Web Text))

Max Activating Token Index: 504

Text #5

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.6020. Min Act: -0.0000

Data Index: 1591699 (Python Code)

Max Activating Token Index: 391

Text #6

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.2686. Min Act: -0.0000

Data Index: 1414340 (Python Code)

Max Activating Token Index: 763

Text #7

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.0313. Min Act: -0.0000

Data Index: 86651 (C4 (Web Text))

Max Activating Token Index: 520

Text #8

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.9358. Min Act: -0.0000

Data Index: 509155 (C4 (Web Text))

Max Activating Token Index: 437

Text #9

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.9030. Min Act: -0.0000

Data Index: 802657 (C4 (Web Text))

Max Activating Token Index: 568

Text #10

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.9884. Min Act: -0.0000

Data Index: 324508 (C4 (Web Text))

Max Activating Token Index: 346

Text #11

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.8320. Min Act: -0.0000

Data Index: 1387875 (Python Code)

Max Activating Token Index: 259

Text #12

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.8067. Min Act: -0.0000

Data Index: 1403296 (Python Code)

Max Activating Token Index: 672

Text #13

Max Range: 5.2138. Min Range: -5.2138

Max Act: 4.0543. Min Act: -0.0000

Data Index: 1487641 (Python Code)

Max Activating Token Index: 460

Text #14

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.7274. Min Act: -0.0000

Data Index: 1256724 (C4 (Web Text))

Max Activating Token Index: 804

Text #15

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.5548. Min Act: -0.0000

Data Index: 1307552 (C4 (Web Text))

Max Activating Token Index: 408

Text #16

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.6928. Min Act: -0.0000

Data Index: 119928 (C4 (Web Text))

Max Activating Token Index: 977

Text #17

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.5048. Min Act: -0.0000

Data Index: 1466883 (Python Code)

Max Activating Token Index: 62

Text #18

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.6626. Min Act: -0.0000

Data Index: 947185 (C4 (Web Text))

Max Activating Token Index: 445

Text #19

Max Range: 5.2138. Min Range: -5.2138

Max Act: 3.8124. Min Act: -0.0000

Data Index: 703639 (C4 (Web Text))

Max Activating Token Index: 712