Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2094 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3526. Min Range: -1.3526

Max Act: 1.3526. Min Act: -0.0000

Data Index: 1145967 (C4 (Web Text))

Max Activating Token Index: 541

Text #1

Max Range: 1.3526. Min Range: -1.3526

Max Act: 1.0296. Min Act: -0.0000

Data Index: 897504 (C4 (Web Text))

Max Activating Token Index: 266

Text #2

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.9716. Min Act: -0.0000

Data Index: 1293617 (C4 (Web Text))

Max Activating Token Index: 222

Text #3

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.9945. Min Act: -0.0000

Data Index: 489812 (C4 (Web Text))

Max Activating Token Index: 792

Text #4

Max Range: 1.3526. Min Range: -1.3526

Max Act: 1.0268. Min Act: -0.0000

Data Index: 1272896 (C4 (Web Text))

Max Activating Token Index: 872

Text #5

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.9040. Min Act: -0.0000

Data Index: 545753 (C4 (Web Text))

Max Activating Token Index: 776

Text #6

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.9203. Min Act: -0.0000

Data Index: 980679 (C4 (Web Text))

Max Activating Token Index: 24

Text #7

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.9162. Min Act: -0.0000

Data Index: 844200 (C4 (Web Text))

Max Activating Token Index: 1023

Text #8

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8348. Min Act: -0.0000

Data Index: 1264211 (C4 (Web Text))

Max Activating Token Index: 502

Text #9

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8062. Min Act: -0.0000

Data Index: 1329348 (C4 (Web Text))

Max Activating Token Index: 347

Text #10

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8360. Min Act: -0.0000

Data Index: 966790 (C4 (Web Text))

Max Activating Token Index: 199

Text #11

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8873. Min Act: -0.0000

Data Index: 210563 (C4 (Web Text))

Max Activating Token Index: 216

Text #12

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8127. Min Act: -0.0000

Data Index: 1058260 (C4 (Web Text))

Max Activating Token Index: 103

Text #13

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8437. Min Act: -0.0000

Data Index: 1282937 (C4 (Web Text))

Max Activating Token Index: 74

Text #14

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8156. Min Act: -0.0000

Data Index: 117978 (C4 (Web Text))

Max Activating Token Index: 159

Text #15

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.7931. Min Act: -0.0000

Data Index: 744579 (C4 (Web Text))

Max Activating Token Index: 129

Text #16

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8335. Min Act: -0.0000

Data Index: 413438 (C4 (Web Text))

Max Activating Token Index: 781

Text #17

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.8045. Min Act: -0.0000

Data Index: 1028664 (C4 (Web Text))

Max Activating Token Index: 767

Text #18

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.7557. Min Act: -0.0000

Data Index: 1057047 (C4 (Web Text))

Max Activating Token Index: 810

Text #19

Max Range: 1.3526. Min Range: -1.3526

Max Act: 0.7965. Min Act: -0.0000

Data Index: 1303961 (C4 (Web Text))

Max Activating Token Index: 104