Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4575 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1646. Min Act: -0.0002

Data Index: 510067 (C4 (Web Text))

Max Activating Token Index: 225

Text #1

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1665. Min Act: -0.0002

Data Index: 533926 (C4 (Web Text))

Max Activating Token Index: 263

Text #2

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1604. Min Act: -0.0002

Data Index: 41713 (C4 (Web Text))

Max Activating Token Index: 27

Text #3

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1542. Min Act: -0.0002

Data Index: 1278835 (C4 (Web Text))

Max Activating Token Index: 96

Text #4

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1564. Min Act: -0.0002

Data Index: 785938 (C4 (Web Text))

Max Activating Token Index: 806

Text #5

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1522. Min Act: -0.0002

Data Index: 1029473 (C4 (Web Text))

Max Activating Token Index: 761

Text #6

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1515. Min Act: -0.0003

Data Index: 852756 (C4 (Web Text))

Max Activating Token Index: 47

Text #7

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1501. Min Act: -0.0002

Data Index: 1120881 (C4 (Web Text))

Max Activating Token Index: 127

Text #8

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1483. Min Act: -0.0002

Data Index: 1254006 (C4 (Web Text))

Max Activating Token Index: 241

Text #9

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1475. Min Act: -0.0003

Data Index: 1323428 (C4 (Web Text))

Max Activating Token Index: 909

Text #10

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1507. Min Act: -0.0002

Data Index: 306120 (C4 (Web Text))

Max Activating Token Index: 976

Text #11

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1435. Min Act: -0.0002

Data Index: 628374 (C4 (Web Text))

Max Activating Token Index: 106

Text #12

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1469. Min Act: -0.0002

Data Index: 365009 (C4 (Web Text))

Max Activating Token Index: 665

Text #13

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1435. Min Act: -0.0002

Data Index: 203347 (C4 (Web Text))

Max Activating Token Index: 658

Text #14

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1426. Min Act: -0.0002

Data Index: 128792 (C4 (Web Text))

Max Activating Token Index: 85

Text #15

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1451. Min Act: -0.0002

Data Index: 703762 (C4 (Web Text))

Max Activating Token Index: 625

Text #16

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1462. Min Act: -0.0002

Data Index: 456726 (C4 (Web Text))

Max Activating Token Index: 27

Text #17

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1439. Min Act: -0.0002

Data Index: 1071735 (C4 (Web Text))

Max Activating Token Index: 413

Text #18

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1431. Min Act: -0.0003

Data Index: 950838 (C4 (Web Text))

Max Activating Token Index: 843

Text #19

Max Range: 0.1665. Min Range: -0.1665

Max Act: 0.1412. Min Act: -0.0003

Data Index: 889309 (C4 (Web Text))

Max Activating Token Index: 579