Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4974 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0457. Min Act: -0.0003

Data Index: 270227 (C4 (Web Text))

Max Activating Token Index: 1007

Text #1

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0432. Min Act: -0.0002

Data Index: 1032364 (C4 (Web Text))

Max Activating Token Index: 805

Text #2

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0425. Min Act: -0.0003

Data Index: 1295714 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0434. Min Act: -0.0002

Data Index: 398726 (C4 (Web Text))

Max Activating Token Index: 665

Text #4

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0406. Min Act: -0.0002

Data Index: 1337068 (C4 (Web Text))

Max Activating Token Index: 488

Text #5

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0406. Min Act: -0.0002

Data Index: 875071 (C4 (Web Text))

Max Activating Token Index: 546

Text #6

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0407. Min Act: -0.0002

Data Index: 1359038 (C4 (Web Text))

Max Activating Token Index: 713

Text #7

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0370. Min Act: -0.0002

Data Index: 86683 (C4 (Web Text))

Max Activating Token Index: 288

Text #8

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0365. Min Act: -0.0002

Data Index: 520840 (C4 (Web Text))

Max Activating Token Index: 712

Text #9

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0367. Min Act: -0.0002

Data Index: 909094 (C4 (Web Text))

Max Activating Token Index: 1014

Text #10

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0364. Min Act: -0.0002

Data Index: 621326 (C4 (Web Text))

Max Activating Token Index: 332

Text #11

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0364. Min Act: -0.0002

Data Index: 837579 (C4 (Web Text))

Max Activating Token Index: 411

Text #12

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0366. Min Act: -0.0002

Data Index: 304818 (C4 (Web Text))

Max Activating Token Index: 707

Text #13

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0347. Min Act: -0.0002

Data Index: 947751 (C4 (Web Text))

Max Activating Token Index: 712

Text #14

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0341. Min Act: -0.0002

Data Index: 1358079 (C4 (Web Text))

Max Activating Token Index: 760

Text #15

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0342. Min Act: -0.0002

Data Index: 160456 (C4 (Web Text))

Max Activating Token Index: 408

Text #16

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0336. Min Act: -0.0002

Data Index: 1101030 (C4 (Web Text))

Max Activating Token Index: 680

Text #17

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0341. Min Act: -0.0003

Data Index: 636520 (C4 (Web Text))

Max Activating Token Index: 76

Text #18

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0333. Min Act: -0.0002

Data Index: 1254980 (C4 (Web Text))

Max Activating Token Index: 447

Text #19

Max Range: 0.0457. Min Range: -0.0457

Max Act: 0.0329. Min Act: -0.0003

Data Index: 1342892 (C4 (Web Text))

Max Activating Token Index: 377