Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1885 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1891. Min Act: -0.0000

Data Index: 1099328 (C4 (Web Text))

Max Activating Token Index: 367

Text #1

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.2023. Min Act: -0.0000

Data Index: 678005 (C4 (Web Text))

Max Activating Token Index: 625

Text #2

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1632. Min Act: -0.0000

Data Index: 711569 (C4 (Web Text))

Max Activating Token Index: 951

Text #3

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1515. Min Act: -0.0000

Data Index: 89026 (C4 (Web Text))

Max Activating Token Index: 672

Text #4

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1633. Min Act: -0.0000

Data Index: 1083680 (C4 (Web Text))

Max Activating Token Index: 779

Text #5

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1627. Min Act: -0.0000

Data Index: 972201 (C4 (Web Text))

Max Activating Token Index: 673

Text #6

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1690. Min Act: -0.0000

Data Index: 501351 (C4 (Web Text))

Max Activating Token Index: 727

Text #7

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1644. Min Act: -0.0000

Data Index: 423515 (C4 (Web Text))

Max Activating Token Index: 425

Text #8

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1565. Min Act: -0.0000

Data Index: 12526 (C4 (Web Text))

Max Activating Token Index: 784

Text #9

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1585. Min Act: -0.0000

Data Index: 40782 (C4 (Web Text))

Max Activating Token Index: 885

Text #10

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1544. Min Act: -0.0000

Data Index: 512548 (C4 (Web Text))

Max Activating Token Index: 974

Text #11

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1509. Min Act: -0.0000

Data Index: 374559 (C4 (Web Text))

Max Activating Token Index: 439

Text #12

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1522. Min Act: -0.0000

Data Index: 431677 (C4 (Web Text))

Max Activating Token Index: 920

Text #13

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1609. Min Act: -0.0000

Data Index: 691892 (C4 (Web Text))

Max Activating Token Index: 740

Text #14

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1470. Min Act: -0.0000

Data Index: 415947 (C4 (Web Text))

Max Activating Token Index: 182

Text #15

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1524. Min Act: -0.0000

Data Index: 488005 (C4 (Web Text))

Max Activating Token Index: 556

Text #16

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1538. Min Act: -0.0000

Data Index: 943151 (C4 (Web Text))

Max Activating Token Index: 340

Text #17

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1447. Min Act: -0.0000

Data Index: 617435 (C4 (Web Text))

Max Activating Token Index: 1001

Text #18

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1386. Min Act: -0.0000

Data Index: 333239 (C4 (Web Text))

Max Activating Token Index: 302

Text #19

Max Range: 0.2023. Min Range: -0.2023

Max Act: 0.1308. Min Act: -0.0000

Data Index: 385902 (C4 (Web Text))

Max Activating Token Index: 904