Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 722 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0303. Min Range: -1.0303

Max Act: 1.0303. Min Act: -0.0000

Data Index: 1326289 (C4 (Web Text))

Max Activating Token Index: 876

Text #1

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.9286. Min Act: -0.0000

Data Index: 1091197 (C4 (Web Text))

Max Activating Token Index: 497

Text #2

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8537. Min Act: -0.0000

Data Index: 259242 (C4 (Web Text))

Max Activating Token Index: 421

Text #3

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8735. Min Act: -0.0000

Data Index: 1277239 (C4 (Web Text))

Max Activating Token Index: 497

Text #4

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8484. Min Act: -0.0000

Data Index: 562602 (C4 (Web Text))

Max Activating Token Index: 228

Text #5

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7895. Min Act: -0.0000

Data Index: 917325 (C4 (Web Text))

Max Activating Token Index: 957

Text #6

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8182. Min Act: -0.0000

Data Index: 1167187 (C4 (Web Text))

Max Activating Token Index: 470

Text #7

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8037. Min Act: -0.0000

Data Index: 512 (C4 (Web Text))

Max Activating Token Index: 608

Text #8

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7785. Min Act: -0.0000

Data Index: 207701 (C4 (Web Text))

Max Activating Token Index: 330

Text #9

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.8271. Min Act: 0.0000

Data Index: 1129757 (C4 (Web Text))

Max Activating Token Index: 113

Text #10

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7575. Min Act: -0.0000

Data Index: 711707 (C4 (Web Text))

Max Activating Token Index: 183

Text #11

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7452. Min Act: -0.0000

Data Index: 1272240 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7492. Min Act: -0.0000

Data Index: 590864 (C4 (Web Text))

Max Activating Token Index: 817

Text #13

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7314. Min Act: -0.0000

Data Index: 1350318 (C4 (Web Text))

Max Activating Token Index: 338

Text #14

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7904. Min Act: -0.0000

Data Index: 541175 (C4 (Web Text))

Max Activating Token Index: 892

Text #15

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7761. Min Act: -0.0000

Data Index: 1216216 (C4 (Web Text))

Max Activating Token Index: 115

Text #16

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7296. Min Act: -0.0000

Data Index: 609667 (C4 (Web Text))

Max Activating Token Index: 238

Text #17

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.6667. Min Act: -0.0000

Data Index: 307462 (C4 (Web Text))

Max Activating Token Index: 355

Text #18

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.7126. Min Act: 0.0000

Data Index: 1330325 (C4 (Web Text))

Max Activating Token Index: 924

Text #19

Max Range: 1.0303. Min Range: -1.0303

Max Act: 0.6682. Min Act: -0.0000

Data Index: 164741 (C4 (Web Text))

Max Activating Token Index: 259