Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2904 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.3036. Min Act: -0.0000

Data Index: 853348 (C4 (Web Text))

Max Activating Token Index: 706

Text #1

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.7196. Min Act: -0.0000

Data Index: 648649 (C4 (Web Text))

Max Activating Token Index: 144

Text #2

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.2643. Min Act: -0.0000

Data Index: 134433 (C4 (Web Text))

Max Activating Token Index: 323

Text #3

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.9973. Min Act: -0.0000

Data Index: 536349 (C4 (Web Text))

Max Activating Token Index: 825

Text #4

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.3129. Min Act: -0.0000

Data Index: 382162 (C4 (Web Text))

Max Activating Token Index: 496

Text #5

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.1264. Min Act: -0.0000

Data Index: 62258 (C4 (Web Text))

Max Activating Token Index: 282

Text #6

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.9994. Min Act: -0.0000

Data Index: 745365 (C4 (Web Text))

Max Activating Token Index: 542

Text #7

Max Range: 7.7196. Min Range: -7.7196

Max Act: 7.2313. Min Act: -0.0000

Data Index: 443826 (C4 (Web Text))

Max Activating Token Index: 472

Text #8

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.6666. Min Act: -0.0000

Data Index: 931072 (C4 (Web Text))

Max Activating Token Index: 868

Text #9

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.7574. Min Act: -0.0000

Data Index: 984180 (C4 (Web Text))

Max Activating Token Index: 68

Text #10

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.5648. Min Act: -0.0000

Data Index: 1003165 (C4 (Web Text))

Max Activating Token Index: 246

Text #11

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.9290. Min Act: -0.0000

Data Index: 411978 (C4 (Web Text))

Max Activating Token Index: 546

Text #12

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.8420. Min Act: -0.0000

Data Index: 93202 (C4 (Web Text))

Max Activating Token Index: 366

Text #13

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.8224. Min Act: -0.0000

Data Index: 775780 (C4 (Web Text))

Max Activating Token Index: 222

Text #14

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.5253. Min Act: -0.0000

Data Index: 88168 (C4 (Web Text))

Max Activating Token Index: 519

Text #15

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.6795. Min Act: -0.0000

Data Index: 1119423 (C4 (Web Text))

Max Activating Token Index: 901

Text #16

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.0177. Min Act: -0.0000

Data Index: 1162109 (C4 (Web Text))

Max Activating Token Index: 456

Text #17

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.0160. Min Act: -0.0000

Data Index: 1337367 (C4 (Web Text))

Max Activating Token Index: 157

Text #18

Max Range: 7.7196. Min Range: -7.7196

Max Act: 6.1199. Min Act: -0.0000

Data Index: 604145 (C4 (Web Text))

Max Activating Token Index: 593

Text #19

Max Range: 7.7196. Min Range: -7.7196

Max Act: 5.9092. Min Act: -0.0000

Data Index: 79621 (C4 (Web Text))

Max Activating Token Index: 198