Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2855 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1193. Min Act: -0.0001

Data Index: 546457 (C4 (Web Text))

Max Activating Token Index: 786

Text #1

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1163. Min Act: -0.0001

Data Index: 786679 (C4 (Web Text))

Max Activating Token Index: 618

Text #2

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1204. Min Act: -0.0002

Data Index: 1223863 (C4 (Web Text))

Max Activating Token Index: 38

Text #3

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1076. Min Act: -0.0001

Data Index: 654096 (C4 (Web Text))

Max Activating Token Index: 857

Text #4

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1180. Min Act: -0.0001

Data Index: 995041 (C4 (Web Text))

Max Activating Token Index: 963

Text #5

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0896. Min Act: -0.0002

Data Index: 1334118 (C4 (Web Text))

Max Activating Token Index: 923

Text #6

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0968. Min Act: -0.0001

Data Index: 453805 (C4 (Web Text))

Max Activating Token Index: 942

Text #7

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0997. Min Act: -0.0001

Data Index: 691011 (C4 (Web Text))

Max Activating Token Index: 16

Text #8

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1015. Min Act: -0.0002

Data Index: 941222 (C4 (Web Text))

Max Activating Token Index: 702

Text #9

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0871. Min Act: -0.0002

Data Index: 419456 (C4 (Web Text))

Max Activating Token Index: 208

Text #10

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1009. Min Act: -0.0002

Data Index: 112725 (C4 (Web Text))

Max Activating Token Index: 174

Text #11

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0892. Min Act: -0.0002

Data Index: 483346 (C4 (Web Text))

Max Activating Token Index: 463

Text #12

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0946. Min Act: -0.0001

Data Index: 559240 (C4 (Web Text))

Max Activating Token Index: 852

Text #13

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0889. Min Act: -0.0002

Data Index: 100860 (C4 (Web Text))

Max Activating Token Index: 585

Text #14

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1003. Min Act: -0.0002

Data Index: 1262851 (C4 (Web Text))

Max Activating Token Index: 725

Text #15

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1024. Min Act: -0.0001

Data Index: 741 (C4 (Web Text))

Max Activating Token Index: 763

Text #16

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1002. Min Act: -0.0001

Data Index: 1143922 (C4 (Web Text))

Max Activating Token Index: 738

Text #17

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0979. Min Act: -0.0002

Data Index: 894537 (C4 (Web Text))

Max Activating Token Index: 21

Text #18

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.0917. Min Act: -0.0002

Data Index: 1078012 (C4 (Web Text))

Max Activating Token Index: 147

Text #19

Max Range: 0.1204. Min Range: -0.1204

Max Act: 0.1040. Min Act: -0.0001

Data Index: 1042508 (C4 (Web Text))

Max Activating Token Index: 350