Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1816 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3782. Min Range: -2.3782

Max Act: 2.3782. Min Act: -0.0000

Data Index: 259315 (C4 (Web Text))

Max Activating Token Index: 655

Text #1

Max Range: 2.3782. Min Range: -2.3782

Max Act: 2.2608. Min Act: -0.0000

Data Index: 1352269 (C4 (Web Text))

Max Activating Token Index: 181

Text #2

Max Range: 2.3782. Min Range: -2.3782

Max Act: 2.0005. Min Act: -0.0000

Data Index: 964376 (C4 (Web Text))

Max Activating Token Index: 675

Text #3

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.8443. Min Act: -0.0000

Data Index: 120768 (C4 (Web Text))

Max Activating Token Index: 971

Text #4

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.9363. Min Act: -0.0000

Data Index: 842220 (C4 (Web Text))

Max Activating Token Index: 518

Text #5

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7774. Min Act: -0.0000

Data Index: 1336602 (C4 (Web Text))

Max Activating Token Index: 442

Text #6

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7741. Min Act: -0.0000

Data Index: 435042 (C4 (Web Text))

Max Activating Token Index: 223

Text #7

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.9352. Min Act: -0.0000

Data Index: 151493 (C4 (Web Text))

Max Activating Token Index: 911

Text #8

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.9084. Min Act: -0.0000

Data Index: 373827 (C4 (Web Text))

Max Activating Token Index: 427

Text #9

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7950. Min Act: -0.0000

Data Index: 395018 (C4 (Web Text))

Max Activating Token Index: 524

Text #10

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.6286. Min Act: -0.0000

Data Index: 949977 (C4 (Web Text))

Max Activating Token Index: 873

Text #11

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7303. Min Act: -0.0000

Data Index: 1068443 (C4 (Web Text))

Max Activating Token Index: 30

Text #12

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7405. Min Act: -0.0000

Data Index: 1194653 (C4 (Web Text))

Max Activating Token Index: 683

Text #13

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.8100. Min Act: -0.0000

Data Index: 908978 (C4 (Web Text))

Max Activating Token Index: 996

Text #14

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.8098. Min Act: -0.0000

Data Index: 348090 (C4 (Web Text))

Max Activating Token Index: 497

Text #15

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.5964. Min Act: -0.0000

Data Index: 274663 (C4 (Web Text))

Max Activating Token Index: 869

Text #16

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.7820. Min Act: -0.0000

Data Index: 441234 (C4 (Web Text))

Max Activating Token Index: 387

Text #17

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.4085. Min Act: -0.0000

Data Index: 589503 (C4 (Web Text))

Max Activating Token Index: 121

Text #18

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.5525. Min Act: -0.0000

Data Index: 702859 (C4 (Web Text))

Max Activating Token Index: 600

Text #19

Max Range: 2.3782. Min Range: -2.3782

Max Act: 1.6384. Min Act: -0.0000

Data Index: 844746 (C4 (Web Text))

Max Activating Token Index: 248