Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2448 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.9212. Min Act: -0.0000

Data Index: 371205 (C4 (Web Text))

Max Activating Token Index: 818

Text #1

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.8904. Min Act: -0.0000

Data Index: 624627 (C4 (Web Text))

Max Activating Token Index: 677

Text #2

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.8619. Min Act: -0.0000

Data Index: 1107212 (C4 (Web Text))

Max Activating Token Index: 499

Text #3

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.8306. Min Act: -0.0000

Data Index: 519827 (C4 (Web Text))

Max Activating Token Index: 518

Text #4

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7925. Min Act: -0.0000

Data Index: 403167 (C4 (Web Text))

Max Activating Token Index: 932

Text #5

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.8064. Min Act: -0.0000

Data Index: 1194548 (C4 (Web Text))

Max Activating Token Index: 631

Text #6

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7424. Min Act: -0.0000

Data Index: 163223 (C4 (Web Text))

Max Activating Token Index: 440

Text #7

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7228. Min Act: -0.0000

Data Index: 158561 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7662. Min Act: -0.0000

Data Index: 992816 (C4 (Web Text))

Max Activating Token Index: 913

Text #9

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7196. Min Act: 0.0000

Data Index: 23995 (C4 (Web Text))

Max Activating Token Index: 476

Text #10

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7168. Min Act: -0.0000

Data Index: 1356932 (C4 (Web Text))

Max Activating Token Index: 301

Text #11

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.7142. Min Act: -0.0000

Data Index: 1064910 (C4 (Web Text))

Max Activating Token Index: 607

Text #12

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6625. Min Act: -0.0000

Data Index: 744399 (C4 (Web Text))

Max Activating Token Index: 143

Text #13

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6506. Min Act: 0.0000

Data Index: 303661 (C4 (Web Text))

Max Activating Token Index: 397

Text #14

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6955. Min Act: -0.0000

Data Index: 1293291 (C4 (Web Text))

Max Activating Token Index: 906

Text #15

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6309. Min Act: -0.0000

Data Index: 1057277 (C4 (Web Text))

Max Activating Token Index: 529

Text #16

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6330. Min Act: 0.0000

Data Index: 952564 (C4 (Web Text))

Max Activating Token Index: 467

Text #17

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6053. Min Act: -0.0000

Data Index: 9112 (C4 (Web Text))

Max Activating Token Index: 144

Text #18

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6453. Min Act: -0.0000

Data Index: 453903 (C4 (Web Text))

Max Activating Token Index: 565

Text #19

Max Range: 0.9212. Min Range: -0.9212

Max Act: 0.6139. Min Act: 0.0000

Data Index: 817560 (C4 (Web Text))

Max Activating Token Index: 669